Methods and apparatus for integrating external applications into an MPEG-4 scene

ABSTRACT

A method of decoding, composing and rendering a scene. First information is obtained, the first information including a part of a MPEG-4 BIFS scene description stream and at least one coded MPEG-4 media stream. The first information is decoded by invoking a BIFS scene decoder and one or more specific media decoders that are required by the scene. Second information is obtained, the second information including a second part of a BIFS scene description stream that contains a reference to an external application. The second information is decoded by invoking the BIFS scene decoder and an external application decoder. An integrated scene is composed, the integrated scene including one or more decoded MPEG-4 media objects and one or more external application objects specified in the decoded scene descriptions streams. The composed integrated scene is rendered on a display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/325,822, filed Dec. 1, 2008, which is a continuation of U.S. patentapplication Ser. No. 11/170,996, filed Jun. 29, 2005, now U.S. Pat. No.7,461,330, filed Dec. 2, 2008, which is a continuation of Ser. No.09/613,292, filed Jul. 10, 2000, now U.S. Pat. No. 6,934,906, filed Aug.23, 2005, which claims priority from U.S. Provisional Application Ser.No. 60/142,745, filed on Jul. 8, 1999, and U.S. Provisional ApplicationSer. No. 60/156,441, filed on Sep. 28, 1999, the contents of which areincorporated herein by reference in their entirety.

FIELD OF THE INVENTION

The present invention relates to the field of digital multimediacompression technology. More particularly, the present invention relatesto methods and apparatus for integrating an MPEG-4 decoder with aninterface of another application that does not conform to the MPEG-4standard.

BACKGROUND

Various types of data decoding devices and applications generatedigitally based presentations which may be experienced by an individualor group of people. Digitally based presentations may be generated by,for example, digital movie players (e.g., a DVD player), digital musicplayers (e.g., a CD player), World Wide Web browsers, Portable DocumentFormat readers (e.g., ADOBE ACROBAT READER), spread-sheet software,word-processing software, operating system software, and video gameplayers. Such presentations may be multimedia presentations that includesome combination of text, graphics, video, animation, and/or sound. Inaddition, digitally based presentations may be interactive presentationsthat allow users to enter data or commands.

In the above-described systems, coded audiovisual information is used asinput to generate the presentation. In the World Wide Web browser, forexample, browser software (such as INTERNET EXPLORER from MICROSOFTCorp.) executing on a general purpose computer may be used to decodeHTML pages (i.e., audiovisual information) and generate an interactivemultimedia presentation. Typically, the various types of systems requirevarious different coding formats for the audiovisual information. Forexample, a World Wide Web browser (“Web browser”) may not be able tooperate using data that is formatted for a CD player.

The Moving Picture Expert Group (MPEG) of the International Organizationfor Standardization (ISO) has introduced standards for coding ofaudiovisual information. These standards include MP3, MPEG-1, MPEG-2,and MPEG-4. MPEG-4 provides a standard for representing units, known as“media objects,” of aural, visual or audiovisual content. These mediaobjects can be of natural origin (e.g., recorded with a camera ormicrophone) or synthetic origin (e.g., generated with a computer).MPEG-4 specifies a standard for describing the composition of theseobjects and for combining these objects to form audiovisual “scenes.” Adescription of MPEG-4 is set forth in MPEG-4 Standard ISO/IEC, 14496-1,hereby incorporated by reference, as well as in Multimedia Systems,Standards and Networks (Atul Puri and Tsuhan Chen, Editors). An MPEG-4decoder receives a bit-stream of data that was originally authored bycoding media objects according to the MPEG-4 systax. The MPEG-4 decoderthen may present the scene specified in the coded data by, for example,displaying the scene on a computer monitor or a television screen. Thescene could contain text, graphics, video, animation, and/or sound inany combination.

Many systems and applications that generate digitally basedpresentations, such as a Web browser or Portable Document Format reader,do not conform to the MPEG-4 standard. In some cases, the author of ascene might want to include the presentation from a non-MPEG-4application, such as a Web browser, into the scene generated by anMPEG-4 decoder. However, prior to the present invention, MPEG-4 did notprovide a method or apparatus by which the author of the scene couldintegrate an application that does not conform to the MPEG-4 standardinto a scene generated by an MPEG-4 decoder.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide methods and apparatus tointegrate an application that does not conform to the MPEG-4 standard(an “external application”) into a scene generated by an MPEG-4 decoder.An embodiment defines a Binary Format for Scenes (BIFS) node that allowsan external application to exist within the MPEG-4 scene graph. In thisembodiment, the external application controls and renders the windowedregion in the MPEG-4 scene, for which the external application'sinterface is defined, allowing natural user interaction with theapplication. In a further embodiment, the scene node is designated“ApplicationWindow.” The present invention provides a node interface,including data definitions and functionality, for the newly definednode.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a scene rendered from MPEG-4 audiovisualinformation which includes an external application integrated thereinaccording to an embodiment of the present invention.

FIG. 2 is a block diagram of a system for integrating an externalapplication into an MPEG-4 scene according to an embodiment of thepresent invention.

FIG. 3 is another block diagram of a scene rendered from MPEG-4audiovisual information which includes an external applicationintegrated therein according to an embodiment of the present invention.

FIG. 4 is a flow diagram illustrating a method for integrating anexternal application into an MPEG-4 scene according to an embodiment ofthe present invention.

FIG. 5 is a block diagram of the architecture of an MPEG-4 playbacksystem according to an embodiment of the present invention.

DETAILED DESCRIPTION

Embodiments of the present invention provide for the integration of anexternal application into an MPEG-4 scene. The phrase “externalapplication” refers to an application that does not conform to someversion of the MPEG-4 standard, and the phrase “MPEG-4 scene” refers toa scene generated by a decoder that decodes an MPEG-4 conformantbit-stream. According to embodiments of the present invention, theexternal application may be presented in an application window withinthe MPEG-4 scene on a display device. As described further below, adecoding device that operates according to embodiments of the presentinvention may be a general purpose computer, a digital signal processor,or a special purpose device such as an application specific integratedcircuit. The logic that integrates the non-MPEG-4 application with theMPEG-4 decoder may be incorporated for example into softwareinstructions, firmware, or a special purpose hardware circuit. TheMPEG-4 decoder and external application may decode audiovisualinformation that is stored in a local memory, streamed, or downloadedfrom a network. According to embodiments of the present invention, theauthor of the MPEG-4 scene may specify features such as the location andsize of the application window in the scene, the time when theapplication is running, whether the external application runs in thebackground, ext.

FIG. 1 is a block diagram of a scene rendered from MPEG-4 audiovisualinformation which includes an external application integrated thereinaccording to an embodiment of the present invention. FIG. 1 shows adisplay device 100 which may be any type of display device capable ofdisplaying information. In an embodiment, display device 100 is amonitor for a general purpose computer system. In another embodiment,display device 100 is a television screen. In another embodiment,display device 100 is a display for an information appliance.

In FIG. 1, display device 100 is displaying an MPEG-4 scene 150 that wasgenerated by an MPEG-4 decoder. In this embodiment, the MPEG-4 decoderis running on a general purpose computer that uses an operating systemin the MICROSOFT WINDOWS family of operating systems. In thisembodiment, in addition to MPEG-4 scene 150, display device 100 alsocontains information such as a Task Bar that was generated by theMICROSOFT WINDOWS operating system. The MPEG-4 decoder that generatesMPEG-4 scene 150 may be any type of MPEG-4 decoder, such as a modifiedMPEG-4 IM1 reference software implementation.

As shown in FIG. 1, MPEG-4 scene 150 is a page of an electronic travelagency and contains MPEG-4 objects 152 and an application window 155. Asillustrated, the application window 155 is a sub-scene that occupiesapproximately the bottom ⅔ of the MPEG-4 scene 150. Of course, theapplication window 155 may be any size and may be located anywherewithin MPEG-4 scene 150. The objects 152 include a talking head agent aswell as text items for Search, View, and Reserve options. In otherembodiments, the MPEG-4 scene may contain any objects that may bepresent in an MPEG-4 scene.

In an embodiment, MPEG-4 scene 150 is context dependent, and differentinformation is shown depending upon whether the user selects the Search,View or Reserve option. In FIG. 1, the user has selected the Reserveoption, and the application window 155 is controlled by a Web browser(e.g., INERNET EXPLORER) that is displaying a hotel reservation page.According to an embodiment of the present invention, the applicationwindow 155 is integrated into the MPEG-4 scene. Within the MPEG-4decoder, a user may interact with MPEG-4 native objects (e.g., objects152) or with the application running in the application window 155.MPEG-4 native objects may be anything defined by an MPEG-4 node, such asfor example a graphical object or a textual object. In this embodiment,the Web browser is not a full featured Web browser (e.g., does notcontain Back command, Forward command, Bookmark command, ext.), but inother embodiments any type of Web browser may be used. The MPEG-4decoder may change the information displayed in response to the MPEG-4bit-stream or user interaction with MPEG-4 native objects (e.g., bymoving the talking head) or the external application may change theinformation displayed (e.g., by displaying a new HTML page). The presentinvention may be used with any MPEG-4 scene and any type of externalapplication, such as a Virtual Reality Modeling Language (VRML)application, Java applet, ext. The mechanism for converging the MPEG-4scene and the external application interface are discussed below.

FIG. 2 is a block diagram of a system 200 for integrating an externalapplication into an MPEG-4 scene according to an embodiment of thepresent invention. System 200 may be used to decode audiovisualinformation that previously had been coded according to any version ofthe MPEG-4 standard.

In addition, system 200 is capable of running an external application,such as a Web browser. System 200 may be a general purpose computer or aspecial purpose device, such as, for example, a set-top box or aninformation appliance.

As shown in FIG. 2, system 200 contains a processor unit 210 that iscoupled to a memory array 220, a display device 230, an input device240, and a network interface 250. In other embodiments, system 200 maycontain additional devices or may contain a subset of these devices. Theterm “coupled” encompasses a direct connection, an indirect connection,or an indirect communication. Processor unit 210 may be any type ofprocessor capable of executing instructions, such as for example amicroprocessor, micro-controller, or special purpose hardware. Memoryarray 220 may be a single memory device or may be a plurality of memorydevices. Memory array 220 may be any type of computer readable mediumand may be or may include a RAM memory, a ROM memory, or a disk memory.In FIG. 2, input device 240 is shown as a mouse. In other embodiments,other types of input devices, such as a keyboard, may be used instead ofor in addition to input devices 240. Network interface 250 is connectedto network 260. Network 260 may be any information systems networkacross which information may be sent. Examples of the network 260include a wireline telephone network, a wireless telephone network, theInternet, an intranet, a virtual private network, or any combination ofthese devices. Any type of network interfaces, such as a networkinterface card or a modem, may be used as or in addition to networkinterface 250.

Display device 230 may be any type of display device that may displayinformation. In FIG. 2, display device 230 is displaying an MPEG-4 scene250. MPEG-4 scene 250 contains a first object 251 (a turtle) and asecond object 252 (a flower) in addition to an application window 255.MPEG-4 scene 250 may be generated by MPEG-4 decoder instructions 221 asdescribed below. First object 251 and second object 252 may be any typesof media objects conforming to an MPEG-4 compliant bit-stream, decodableby an MPEG-4 decoder, such as text, graphics, video, or animation. In afurther embodiment, MPEG-4 scene 250 may contain more or less objectsthan shown in FIG. 2. Application window 255 is interactive with andrendered by an external application such as a Web browser, PortableDocument Format reader, spreadsheet software, ext.

Memory array 250 may contain MPEG-4 decoder instructions 221,integration instructions 222, scene description information 225 in theform of an MPEG-4 bit-stream, and external application instructions 229.An MPEG-4 stream is information that may be read be an MPEG-4 decoder.The MPEG-4 stream may be information stored or a memory device or may beinformation that is transferred over a network such as the Internet. AnMPEG-4 stream may include an MPEG-4 scene description streams (e.g., acoded version of a scene description that was written in a MPEG-4 scenedescription language) as well as one or more MPEG-4 media streams (e.g.,a coded version of a media object such as MPEG-2 audio, MPEG-2 video,MPEG-4 audio, MPEG-4 video, audio animation, visual animation, ext.). AnMPEG-4 media stream may be decoded by an appropriate specificapplication decoder, such as a video decoder or audio decoder. Thedecoder instructions 221 and integration instructions 222 may be adaptedto be executed by a processor. The phrase “instructions adapted to beexecuted by a processor” is meant to encompass source code, assembler,and any other expression of instructions that may require preprocessingin order to be executed by processor.

Scene description information 225 may be binary audiovisual informationthat has been coded by an MPEG-4 coder. Scene description information225 may describe, for example, MPEG-4 scene 250. MPEG-4 decoderinstructions 221 are a software embodiment of an MPEG-4 decoder asdiscussed above. MPEG-4 decoder instructions 221 may be executed byprocessor unit 210 to decode scene description information 225 andgenerate MPEG-4 scene 250.

External application instructions 229 may be the instructions for anexternal application such as a Web browser, Portable Document Formatreader, spreadsheet software, ext. In this embodiment, externalapplication instructions 229 act as an external application decoder anddecode information that is in a format recognized by the externalapplication. MPEG-4 decoder instructions may contain integrationinstructions 222 for integrating the MPEG-4 decoder with an externalapplication such as external application instructions 229. According tothis embodiment, integration instructions 222 may be invoked when theMPEG-4 decoder instructions 221 encounter a reference to an externalapplication in the scene description information 225 that calls for theintegration of an external application into the MPEG-4 scene. In thisembodiment, the integration instructions 222 may use scene descriptioninformation 225 to configure and start external application instructions229 as discussed below. The external application instructions 229 maythen render the sub-scene in application window 255, may receive userinput from input device 240, and may access network 260.

According to another embodiment of the present invention, the scenedescription information 225 need not be stored in memory 220, but rathermay be a bit-stream of data that is accessed from a network such asnetwork 260. In a further embodiment, the bit-stream of the scenedescription information 225 may be stored in a memory in device 200 asit is streamed. In a further embodiment, the external applicationinstructions 221 need not be stored on memory array 250 but rather maybe accessed via network 260. In a still further embodiment, MPEG-4decoder instructions 221 are not stored on memory array 220 but areaccessed via network 260.

In another embodiment, system 200 contains MPEG-4 decoder logic insteadof MPEG-4 decoder instructions 221. In this embodiment, MPEG-4 decoderlogic may be firmware or a circuit that performs MPEG-4 decoding. MPEG-4decoder logic may be part of a decoder board.

In another embodiment, the reference to an external application in thescene description information that provides for the integration of anexternal application into the MPEG-4 scene is a MPEG-4 BIFS node. BIFSis an MPEG language for scene description. In an embodiment, theposition of the external application, its dimension, whether theapplication is active or not, ext., may be specified through BIFS sceneauthoring. A scene description written in VRML-like text can be used bya MPEG-4 coder to develop a binary coded version (i.e., BIFS) of theaudiovisual information that makes up a scene. BIFS nodes are the scenegraph elements that may be used to describe audio-visual primitives andtheir attributes, along with the structure of the scene graph itself. Inan embodiment, the BIFS node contains information that may be used toconfigure and start the external application. In a further embodiment,the node may be coded and included in a scene description bitstream.

In an embodiment, the node that provides for the integration of anexternal application into the MPEG-4 scene is the “ApplicationWindow”node. When the MPEG-4 decoder encounters an ApplicationWindow node, itmay use the information contained in the node to integrate anapplication into the MPEG-4 scene. That is, the decoder may start theapplication and may create an application window when it encounters theApplicationWindow node. The windowed region may be controlled andrendered by the external application, thus allowing natural userinteraction with the application. The ApplicationWindow node may allowan external application such as a Web browser to exist within the MPEG-4scene graph. In an embodiment, the ApplicationWindow node is anSF2Dnode, according to an MPEG-4 standard. In an embodiment, the nodeinterface of the ApplicationWindow node may be defined as follows:

ApplicationWindow type field name default value { exposedField SFBooIisActive FALSE exposed Field SFTime startTime 0 exposedField SFTimestopTime 0 exposed Field SFString description exposedField MFStringparameter exposedField MFString url 0 exposedField SFVec2f size 0, 0 }

In an embodiment, the MPEG-4 encoder may place designations before eachfield in the ApplicationWindow node which may be used by the decoder todetermine what the information contained in the field is to be used for.In this embodiment, the fields can be written by the scene's author inany order.

The “url” field contains Universal Resource Locator information for theexternal application to be opened. The url field may carry the locationwhere software for the external application resides. The location may bewithin the decoder system (e.g., within a general purpose computer) ormay be accessible over a network such as the Internet. The url field maybe a string. When the ApplicationWindow node gets coded, the url fieldis coded into binary information. When the ApplicationWindow node isdecoded, the decoder may use the url to find the location of theexternal application. In an embodiment, the default value for the urlfield is null. In this embodiment, if a url is not specified, anexternal application will not be integrated.

The “size” field provides the dimension (width and height) of theapplication window, for example in units of pixels. As shown above, thedefault value for the size field is 0, 0. In an embodiment, if a size isnot specified, an application window will not be displayed. In anembodiment, the location of the application window is not specifiedwithin the ApplicationWindow node, but rather (as shown below) isspecified in the parent node. In an embodiment, the location may beprovided relative to a reference location of the application window.

The “parameter” field carries parameters to be interpreted by theapplication decoder when the application window is instantiated. Thisfield may contain any required parameters for starting the application.The parameters may be passed to the external application. As shownabove, the default value for the parameter field is null. In anembodiment, if parameters are not specified, no parameters will bepassed to the external application.

The “description” field allows a prompt to be displayed as analternative to the “url” in the “url” field. In this embodiment, if astring is not specified, a prompt will not be displayed. Thus, thescene's author may select whether or not to display a prompt. As shownabove, the default value may be a blank string.

The “startTime” field indicates when the external application is to bestarted. At the time specified in this field, the application is givencontrol of the rendering window defined by the size field. The“stopTime” field indicates that the application is finished and shouldbe shut down. At the time specified by the stopTime field, the renderingwindow defined by the size field is returned to the MPEG-4 player. In anembodiment, the default value for startTime and stopTime is 0. In thisembodiment, if a startTime is not specified, the external application'sstarting time is not delayed. In a further embodiment, the times may bespecified relative to an MPEG-4 decoder clock.

The “isActive” field signals the application is to relinquish itsrendering window to the MPEG-4 player, but to continue to run. That is,when the isActive filed is specified as “FALSE,” the application runs inthe background. The value of the isActive field for a node may bechanged during the execution of the external application.

In another embodiment, the node may use different designations, havedifferent fields, or have different default values. For example, inanother embodiment, the ApplicationWindow node may have a subset of thefields described above.

An example of the text used to generate BIFS commands that may be usedby a scene author to integrate an application into an MPEG-4 sceneaccording to an embodiment of the present invention is provided below.The code fragment below may be used to generate the MPEG-4 scene shownin FIG. 3 and is described below with reference to FIG. 3. The codefragment below is only a part of the BIFS scene description used togenerate this scene, but it is useful to convey an understanding of theinvention.

Transform2D {     translation −50 150     children [      def APPWIN_1ApplicationWindow {        url “www.att.com”        startTime  0       stopTime  0        isActive  TRUE        size  300, 300        }    ] }

FIG. 3 is a block diagram of a scene rendered from MPEG-4 audiovisualinformation which includes an external application integrated thereinaccording to an embodiment of the present invention. Display device 300contains an MPEG-4 scene 350. The MPEG-4 scene 350 contains three textobjects, START, STOP, and TEST OCCLUDED MPEG-4 OBJECT. The MPEG-4 scene350 also contains an occluded circle object and an application window355.

In the code fragment shown above, the url of the application isspecified as the first page of the AT&T site on the World Wide Web. Whenit encounters this node, an MPEG-4 decoder will start a Web browser thatis directed to the AT&T site. Thus, in FIG. 3, the application window iscontrolled and rendered by a Web browser that is displaying the firstpage of the AT&T site on the World Wide Web. The url may also containthe location of an application, and the url of the page to be displayedmay be a parameter to the node. Because the application window 355 iscontrolled by the Web browser, the user may browse the AT&T site byentering input into the application running within the applicationwindow, and the user's input may be passed to the Web browser withoutinvoking a response from the MPEG-4 decoder. The MPEG-4 decoder hascontrol over the size of the application window and control over whetheror not the application window is being displayed. The decoder may have arendering loop during which it refreshes the objects in an MPEG-4 scene.According to an embodiment, when an external application is displayingan application window in the MPEG-4 scene, the MPEG-4 decoder will notrefresh or impinge upon the region occupied by the application window.

In the code fragment above, the location of the application window isspecified in the translation field of the Transform2d node (−50 150).This location is used to position the application window as shown inFIG. 3. Similarly, the size of the application window is specified as300 pixels by 300 pixels, and this size information is used to determinethe size of the application window in FIG. 3. The startTime and stopTimeare set at 0, so the application will not be delayed and does not have apre-determined stop time. The isActive field is set to TRUE, so theapplication will run in the foreground.

In the example above, the external application controls the portion ofthe display that is occupied by the application window. In thisembodiment, the application window will overlay and occlude any objectsthat impinge upon the application window. Thus, as shown in FIG. 3, theportion of the circle object that would overlay the application windowis hidden behind the application window. If the application window wereto be closed, then the entire circle object would be displayed in theMPEG-4 scene 350.

According to an embodiment, objects such as the text START and STOP ofFIG. 3 may be routed to the startTime and stopTime for the externalapplication node. According to this embodiment, a user may be able tostart the external application by selecting the START object (e.g., byclicking on the object) and may be able to close the externalapplication by selecting the STOP object. In this case, the applicationwindow may be opened when the user starts the external application andmay be closed when the user stops the external application.

FIG. 4 is a flow diagram illustrating a method for rendering a scenethat includes integrating an external application into an MPEG-4 sceneaccording to an embodiment of the present invention. According to thisembodiment, first information that includes a part of a MPEG-4 BIFSscene description stream and at least one coded MPEG-4 media stream isobtained (401). The first information may be obtained, for example, overa network or from a memory device. The first information is decodedusing a BIFS scene decoder and one or more specific application decodersassociated with the scene description (402). A specific applicationdecoder may be, for example, a video decoder that decodes a video mediastream. Second information is obtained including a second part of a BIFSscene description stream that contains a reference to an externalapplication (403). The reference to an external application maycorrespond to an ApplicationWindow node. The second information isdecoded using the BIFS scene decoder and an external application decoder(404). The external application decoder may be an application such as,for example, a web browser or a PDF reader. An integrated sceneincluding one or more decoded MPEG-4 media objects and one or moreexternal application objects is composed (405). The composed integratedscene is rendered, for example, using a display device (406).

In an embodiment, integrating the external application into the scenemay include creating an application window within the scene. Inaddition, integrating the external application into the scene mayinclude configuring the system to provide that the external applicationreceives any input within the application window and controls any outputthrough the application window. In addition, rendering the integratedscene may include preventing native MPEG-4 objects from displayinginformation within the application window.

In an embodiment, the reference to an external application may include acoded representation of a Universal Resource Locator relating to alocation where software (e.g., a decoder) for the external applicationresides. In a still further embodiment, the reference to an externalapplication includes information identifying a location for anapplication window within the scene. In another embodiment, thereference to an external application includes information identifyingdimensions of the application window. In another embodiment, thereference to an external application includes a start time and a stoptime, and the method of rendering a scene further comprises starting theexternal application at the start time and stopping the externalapplication at the stop time. In a still further embodiment, thereference to an external application includes description information,and integrating the external application into the scene includesdisplaying a prompt in the scene based on the description information.In another embodiment, the reference to an external application includesapplication parameters, and starting the external application includespassing the application parameters to the external application.

In another embodiment, a scene is composed and rendered before thereference to an external application is obtained. In this embodiment,second information including a reference to an external application maybe obtained after composing and rendering the scene, and the secondinformation may be decoded using a BIFS scene decoder and an externalapplication decoder. The scene may then be recomposed by integrating asub-scene specified by the decoded second information into the scene,and the re-composed scene may be rendered.

Embodiments of the invention may include a subset of the above describedfeatures or may include additional features, and the order of the stepsdescribed may be varied.

FIG. 5 is a block diagram of the architecture of an MPEG-4 playbacksystem according to an embodiment of the present invention. In FIG. 5,an MPEG-4 stream is delivered over Network/Storage Medium 510 viaDelivery layer 520. In this embodiment, streams may be obtained bytransmission of a network or from a local storage device. Delivery layer520 may include a transport multiplex, a TransMux (not standardized byMPEG-4 but could be UDP, AAL 2, MPEG-2 Transport Stream, ext.), and anoptional multiplex called the FlexMux. Demultiplexed streams may leavevia an interface (e.g., a DAI interface) and enter the Sync layer 530,resulting in SL packetized elementary streams 531 that are ready to bedecoded. The streams may be decoded in Compression layer 540, which mayencapsulate the functions of the media objects 541, scene description542 and object descriptor 543 decoding. The scene description 542 maycontain a reference to an external application, such as the ApplicationWindow node discussed above. Compression layer 540 may yield individualdecoded objects and related descriptors. The external application maydecode a stream and yield a decoded external object 545. The compositionand rendering process 550 may use the scene description and decodedmedia to compose and render the Audiovisual scene 556, which may includea window for an external application such as Web Application Window 555.Composition may involve putting objects in a scene together, andrendering may involve presenting the scene to the user. Thus, thecomposition and rendering process 555 may pass the audiovisual scene tothe presenter, which may be one or more of the input/output devicesdiscussed above, and may allow a user to interact with the presentationof the scene, as shown in Display and User Interaction 560. The actionsnecessary for user interaction 561 (e.g., a request for additional mediastreams) may be sent back to the Network/Storage Medium 510 through thecompression layer (via Rendering Channel Encoding 554), sync layer anddelivery layer.

MPEG-4 is an object based standard for multimedia coding that codesindividual video objects and audio objects in the scene and delivers inaddition a coded description of the scene. In the present invention, thecoded scene description may include a reference to an externalapplication. At the decoding end, the scene description and individualmedia objects are decoded, synchronized and composed for presentation.In the present invention, an external application decoder may decodeinformation into one or more external application objects.

Several embodiments of the present invention are specificallyillustrated and/or described herein. However, it will be appreciatedthat modifications and variations of the present invention are coveredby the above teachings and within the purview of the appended claimswithout departing from the spirit and intended scope of the invention.For example, while FIG. 2 shows MPEG-4 decoder instructions stored in amemory array, the MPEG-4 decoder could be implemented in hardware (e.g.,a decoder board) or downloaded from a network. As another example, thefields of the node described may be varied.

We claim:
 1. A method comprising: decoding, via a binary format scenedecoder, a part of a binary format scene description stream thatreferences a non-MPEG external application object and a pointer to a setof non-MPEG computer-executable instructions associated with thenon-MPEG external application object to yield decoded data, wherein thenon-MPEG external application object is configured to control and rendera windowed region within a coded scene according to the set of non-MPEGcomputer-executable instructions; and composing, based on the decodeddata, an integrated scene comprising the non-MPEG external applicationobject.
 2. The method of claim 1, further comprising decoding firstinformation from the binary format scene description stream using thebinary format scene decoder and a specific application decoderassociated with a scene description.
 3. The method of claim 2, furthercomprising decoding second information comprising the part of the binaryformat scene description stream using the binary format scene decoderand an external application decoder.
 4. The method of claim 1, whereinthe binary format scene description stream conforms to an industrystandard.
 5. The method of claim 4, wherein the industry standard isMPEG-4.
 6. The method of claim 1, wherein the pointer to the set ofnon-MPEG computer-executable instructions software associated with thenon-MPEG external application object further comprises a codedrepresentation of a universal resource locator.
 7. The method of claim1, wherein the non-MPEG external application object is associated withan external application.
 8. The method of claim 7, wherein the set ofnon-MPEG computer-executable instructions reside on a servercommunicating over a network with a web browser.
 9. The method of claim7, wherein the external application is a web browser.
 10. The method ofclaim 9, further comprising displaying a prompt in the integrated scenebased on description information.
 11. The method of claim 7, furthercomprising receiving a start time and a stop time associated with theexternal application.
 12. The method of claim 11, further comprising:starting an external application decoder at the start time; and stoppingthe external application decoder at the stop time.
 13. The method ofclaim 7, wherein a reference to the external application furthercomprises information corresponding to an ApplicationWindow node. 14.The method of claim 13, wherein the information corresponding to theApplicationWindow node further identifies a location for an applicationwindow within the integrated scene and identifies dimensions of theapplication window.
 15. The method of claim 14, wherein the informationcorresponding to the ApplicationWindow node further comprises a starttime and a stop time.
 16. The method of claim 1, further comprising:re-composing a new integrated scene by integrating a sub-scene specifiedby the part of the binary format scene description stream into theintegrated scene; and rendering the new integrated scene.
 17. The methodof claim 16, further comprising: starting the non-MPEG externalapplication object at the start time; and stopping the non-MPEG externalapplication object at the stop time.
 18. A system comprising: aprocessor; and a computer-readable storage medium having instructionsstored which, when executed by the processor, cause the processor toperform operations comprising: decoding, via a binary format scenedecoder, a part of a binary format scene description stream thatreferences a non-MPEG external application object and a pointer to a setof non-MPEG computer-executable instructions associated with thenon-MPEG external application object to yield decoded data, wherein thenon-MPEG external application object is configured to control and rendera windowed region within a coded scene according to the set of non-MPEGcomputer-executable instructions; and composing, based on the decodeddata, an integrated scene comprising the non-MPEG external applicationobject.
 19. The system of claim 18, the computer-readable storage mediumhaving additional instructions stored which result in the operationsfurther comprising decoding first information from the binary formatscene description stream using the binary format scene decoder and aspecific application decoder associated with a scene description.
 20. Anon-transitory computer-readable storage device having instructionsstored which, when executed by a computing device, cause the computingdevice to perform operations comprising: decoding, via a binary formatscene decoder, a part of a binary format scene description stream thatreferences a non-MPEG external application object and a pointer to a setof non-MPEG computer-executable instructions associated with thenon-MPEG external application object to yield decoded data, wherein thenon-MPEG external application object is configured to control and rendera windowed region within a coded scene according to the set of non-MPEGcomputer-executable instructions; and composing, based on the decodeddata, an integrated scene comprising the non-MPEG external applicationobject.